Evaluation should be a key source of information for budget decision-makers. In practice, however, it has had surprisingly little impact on decisions about what to fund and by how much1. This raises the question: what can be done to realize the true potential of evaluation to improve budgeting?
The failure of evaluation to contribute much to resource allocation decisions has been on particularly vivid display during episodes where governments have set out to make major spending cuts (e.g. Canada in the 1990s and the UK after the global financial crisis). During these episodes, available evaluation reports proved to be of little assistance in guiding difficult decisions about where to cut spending.
The problem has, however, been a more general one, with finance ministries repeatedly complaining that evaluations are not very useful for their purposes. Indeed, the poor track record of evaluation in supporting budgeting has been one factor contributing to rise of spending review.2
This is the third in a 3-part series on evaluation. The first focused on the importance of practical evaluation methods, and the second on the need for more systematic efficiency analysis.
Spending review is not a form of evaluation. Evaluation means the application of formal analytic methods (such as impact evaluation and contribution analysis/program logic analysis)3. Spending reviews, by contrast, have no prescribed methodologies and use whatever formal and informal analytic approaches make sense for the topic and are feasible within the time available. Spending reviews are carried out quickly, to provide timely input into budget preparation. Most evaluation methods take considerably more time.
Evaluation should, however, be an important part of the information base for spending review. Although most evaluation takes too much time to be carried out in conjunction with a spending review, spending reviews should be able to make extensive use of evaluations carried out previously. In addition, it should be possible to use certain of the more practical evaluation methods during spending reviews in relevant cases. The better the information base that spending review can draw on – including from evaluation – the better it will do its core job of presenting savings options to budget decision-makers.
More on how evaluation should support spending review may be found in my presentation at a workshop organized by the Brazilian Center for Learning on Evaluation and Results at the Getulio Vargas Foundation as part of the 2024 gLocal Evaluation Week.
Well, that’s the theory. In practice, spending review doesn’t seem to make much use of evaluation. Although they should in principle work closely together, spending review and evaluation are essentially separate processes.
How then can evaluation be made more useful to support spending reviews and budgeting generally? In considering this question, several points stand out.
The first point is that we need to increase the amount of systematic efficiency analysis. Efficiency analysis – that is, analysis aimed at identifying ways of lowering the costs of delivering government outputs – is particularly useful for budgeting. However, as discussed in my last blog piece, evaluation is not much good at efficiency analysis, even though it claims to cover both effectiveness and efficiency. But we shouldn’t get hung up on whether we call efficiency analysis “evaluation” or not. The important thing is to put more effort into the application of methods such as cost analysis and business process analysis. Doing so will yield big dividends for budgeting.
The second point is about how to make evaluations of effectiveness more useful for budgeting. Assessing effectiveness is what evaluation does best. But most effectiveness evaluations are not useful for budgeters. What they are most useful for is helping agencies to make policy and implementation changes that increase the effectiveness of government interventions. This is not to say that evaluations of effectiveness are never useful for budgeting. If an evaluation finds that a program is essentially ineffective, this may be of great relevance to the budget because it may lead to the decision to close it down. However, only a small minority of evaluations conclude that the program being evaluated is ineffective.
It is nevertheless possible to substantially increase the contribution that effectiveness evaluations make to budgeting. The most obvious way of doing this is by initiating more evaluations of programs that are suspected to be ineffective, and which are potential candidates for elimination. But effectiveness evaluations can also make other contributions to budgeting.4
This leads us to the third point, which is about who decides what will be evaluated and what the criteria of the evaluation will be. The reality is that evaluation will never realize its potential as an instrument for informing budgeting unless ministries of finance have more control over the choice of topics and terms of reference of evaluations.
The distinction between self-evaluation and centralized evaluation is helpful here. Self-evaluation means that line ministries initiate evaluations and carry them out themselves. Centralized evaluation means that central agencies such as the ministry of finance initiate and manage evaluations of line ministry policies and programs. In the Anglo-Saxon countries, almost all evaluation is self-evaluation, whereas in certain other countries (e.g. Chile) most evaluation is centralized. One clear lesson from experience is that self-evaluations are in general of limited value for the ministry of finance or other central decision-makers – for reasons that will no doubt be obvious. It follows that if the ministry of finance is serious about making evaluation a tool that works for it, it is necessary to have a program of centralized evaluations under its control.
This is not to say that self-evaluation is a bad thing. If line ministries are serious about it – and not just undertaking evaluations as a compliance exercise – it has the potential to help them substantially improve their performance. The ideal evaluation system combines both self-evaluation and centralized evaluation.
Evaluation has, in most countries, the potential to make a much bigger contribution to budgeting than it does at present.5 But action on multiple fronts will be required to realize that potential.
- As my colleague Jordi Baños-Rovira puts it based on a review of OECD survey data, “public policy evaluation shows a very low level of integration with the government budget, playing a minor role in budget decisions.” (El reto de integrar la presupuestación pública con la evaluación de políticas.) I reached the same conclusion in my 2014 study Connecting Evaluation and Budgeting. ↩︎
- An example of spending review driven by “dissatisfaction” with evaluation – more specifically, with its failure to deliver sufficient “useful information for budget decision-making” is to be found in the Canadian province of Quebec (see Jacob et al, “Evaluation et révision des dépenses publiques” in Praxis de l’évaluation et de la révision des programmes publics). ↩︎
- Most definitions of evaluation state that it involves “systematic” analysis, where systematic refers to the application of formal evaluation methods. ↩︎
- For example, in assessing the extent to which transfer payments and other benefits are correctly targeted – i.e. going to the right people. ↩︎
- Apart from efficiency and effectiveness, there is another evaluation criterion which is significant for budgeting – relevance. Irrrelevant spending should by definition be eliminated. Assessing the relevance of spending is something that can generally be done as part of a spending review without using analytical methods which are part of the evaluation toolkit. ↩︎